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NAVAL  POSTGRADUATE  SCHOOL 
Monterey,  California 

Rear  Admiral  M.  B.  Freeman,  USN  M.  U.  Clauser 

Superintendent  Provost 

ABSTRACT: 

A  study  is  devoted  to  the  development  of  statistical  procedures 
to  be  used  to  test  the  validity  of  the  Ships  Supply  Support  Study  simulator. 
Some  theory  is  presented  for  each  test  procedure,  but  special  emphasis  is 
paid  to  describing  each  test  in  detail  so  that  the  tests  can  be  implemented 
by  the  project  study  group.   Examples  which  illustrate  the  required  numerical 
work  are  given  with  each  test  procedure,  and  the  advantages  and  disadvantages 
of  each  are  pointed  out.   A  computer  program  which  will  perform  the  calcu- 
lations necessary  for  the  more  complicated  test  is  included. 

This  test  was  supported  by  the  Research  and  Development  Division, 
Naval  Supply  Systems  Command,  under  NAVSUP  RDT&E  No.  TF  38.531.001. 

Prepared  by: 


1 .   INTRODUCTION 

A  problem  associated  with  any  simulation  study  is  to  test  the 
results  obtained  from  the  simulator  to  determine  how  well  the  system 
being  studied  is  duplicated.   It  is  only  after  the  simulation  results 

have  been  validated  that  any  reliance  can  be  placed  on  the  output  of 

4 
that  simulator.   In  the  Ships  Supply  Support  Study  (S  )  a  simulator 

is  used  to  obtain  estimates  of  the  mean  supply  response  time  (MSRT) 
for  each  of  13  material  cognizance  classes  (cogs) .   In  addition, 
random  samples  of  actual  response  times  observed  by  ships  in  the 
fleet  for  each  of  the  cogs  have  been  collected.   In  order  to  validate 
the  S   simulation  results  the  simulator  MSRT's  should  be  compared  to 
the  available  data.   Since  cog  response  times  are  random  variables 
with  perhaps  different  distributions  for  each  cog,  statistical  methods 
are  desired  to  test  whether  or  not  the  simulator  MSRT's  are  indeed 
the  same  as  the  means  of  the  actual  cognizance  class  response  times, 
the  populations  in  this  study. 

There  are  several  well  known  statistical  techniques  which 
can  be  used  to  perform  the  above  tests.   This  report  describes  briefly 
some  of  those  tests  and  discusses  some  of  the  advantages  and  dis- 
advantages of  each.   Detailed  instructions  for  implementing  the  tests 
are  presented,  and  examples  are  given  to  illustrate  the  numerical 
work  which  must  be  done. 

Although  the  statistical  tests  are  developed  and  described 

4 
with  reference  to  S  ,  they  should  also  be  useful  for  validating 

future  simulations. 


2.   NOTATION 

The  following  standard  notation  will  be  used  in  describing 
the  statistical  tests  in  this  report.   Let  n.   be  the  number  of 
actual  ship  observations  of  supply  response  time  for  cog  i.   Then, 

f*Vi 

let  X. .   be  the  j     observation  of  supply  response  time  for 
cog  i;   i  =  1,  2,  ...,13   and  j  =  1,  2,  ,.,,  n  .   Let 


i 


n. 

l 

X.  =   J   X. .  /  n. 

1    jii  «      1 

n. 

and  S?  =   7   (X..-X.)2  /  (n.-l), 

i    .L.        ij   i       i 
j-l 

the  sample  means  and  sample  variances,  respectively,  for  cog  i. 
The  value  y.   will  denote  the  actual  (unknown)  population  mean 

supply  response  time  for  cog  i,  and   X.   will  represent  the  value 

4 
obtained  from  the  S   simulator  for  the  MSRT  of  cog  i. 


3.   A  SINGLE  TEST  FOR  THE  MULTIPLE  HYPOTHESIS  TESTING  PROBLEM 

3 . 1   BACKGROUND 

A 
It  is  desirable  to  validate  the  S   simulator  by  testing 

whether  or  not  each  simulator  MSRT  approximates  the  population  MSRT. 

In  this  section,  a  single  test  is  derived  which  will  validate  or 

invalidate  the  entire  set  of  simulator  MSRT's.   Thus,  it  offers  a 

computational  advantage  of  requiring  few  calculations. 

Consider  the  hypothesis  testing  problem: 

H  :  y.  =  X.  i  =  1,  2 13 

oil  ' 

vs 

Hn  :   y,  ^  X.  for  some  i. 

1    i    l 


H   is  the  null  hypothesis,  and  it  simply  states  that  each  of  the 
o 

simulator  MSRT's  is  the  actual  mean  of  the  population  of  the  respec- 
tive cog  response  time.  The  alternative  hypothesis,  1^  states  that 
some  simulator  MSRT  is  not  the  population  MSRT.   If  each  of  the 
sample  sizes  n±,   n2>  ...,  n13   is  sufficiently  large  (50  or  more), 
this  hypothesis  testing  problem  does  not  require  any  assumptions 
about  the  underlying  populations  of  the  supply  response  times  other 
than  requiring  that  the  populations  have  finite  first  and  second 
moments.   This  is  important  because  little  work  has  been  done  in 
determining  the  properties  of  the  response  time  distributions. 

3.2  DERIVATION  OF  THE  TEST  STATISTIC 

With  fairly  large  sample  sizes  the  central  limit  theorem 
indicates  that  the  distribution  of  each  X±  is  approximately 
normal  with  mean  U±  and  variance  0±   /  n±,   regardless  of  the 
population  distribution.   Thus, 


is  approximately  chi-square  distributed  with  one  degree  of  freedom, 
denoted  X^1)-   Furthermore,  the  X. 's   are  independent  so  that 


i-1 


i-    I    z. 


2 

is  chi-square  distributed  with  13  degrees  of  freedom,   X  (13). 

2 
For  the  statistical  problem  at  hand  the  population  variance  0±y 

like  the  population  mean,  is  unknown.   To  circumvent  this  problem 


2 
the  sample  variance  S.   is  substituted  for  the  population  variance 

2 
0    .   This  substitution  should  not  be  serious  if  the  sample  sizes  are 


large, 


When  the  null  hypothesis  is  true;  that  is,   y.  =  X   for  all 


i,  the  distribution  of 

13  (X.-A.)2  n. 
D-  I     -S* X 

i=i       s: 

i 

will  be  approximately  chi-square  with  13  degrees  of  freedom.   On 
calculating  a  value  of   D  based  on  the  observations  in  a  random 

sample,  the  value  should  not  be  large  if  the  null  hypothesis  is  true. 

2 
To  be  more  precise,  let  X  (13)    represent  that  unique  number  such 

that  the  probability  that  an  observation  of  a  random  variable  having 

a  distribution  which  is  chi-square  with  13  degrees  of  freedom  exceeds 

2 
X  (13)   is  a.   Then,  if  the  null  hypothesis  is  true,  an  observed 

2 
value  of  D  would  exceed  the  value  x  (13)  with  a  probability  of 

a.   If  a  is  small,  it  is  an  unlikely  event  that  a  value  of  D 

2 
exceeds   X  (13) .   The  statistician  is  led  to  believe  that  H   is 
Aa  o 

false  whenever  such  an  event  occurs. 


3.3  DESCRIPTION  OF  THE  TEST  PROCEDURE 

The  test  procedure  reduces  to  an  especially  simple  comparison 

2 
of  the  calculated  value  of   D  with  the  number  x  (13) . 

(1)   Determine  the  maximum  probability  that  one  is  willing 

to  reject  H   when  H   is  in  fact  true.   Denote 
J      o         o 

this  probability,  the  level  of  significance  of  the  test, 
by  a. 


2 

(2)  Determine  the  number  XaCl3)   from  standard  chi-square 

tables. 

(3)  Calculate  a  value  of  D  based  on  the  observed  random 

samples. 

2 

(4)  If  the  value  of  D  exceeds  the  number  Y  (13)   reject  H  , 

Aa  o 

3.4  DETERMINING  THE  POWER  OF  THE  STATISTICAL  TEST 

When  the  level  of  significance,   a,  is  chosen  the  probability 

of  rejecting  the  null  hypothesis  when  it  is  true  is  determined-.   It 

is  also  useful  to  determine  the  probability  of  rejecting  the  null 

hypothesis  when  it  is  false,  the  power  of  the  test.   Suppose  y.  =  6. 

and  for  at  least  one  value  of   i,  6.  ^  X  .  .   The  random  variable  D 

l    l 

is  now  distributed  as  a  non-central  chi-square  random  variable  with 
13  d.f.  and  non-centrality  parameter 


13/ „  ^  N2 


B-i  rw" 


i=l 

Tables  for  the  non-central  chi-square  distribution  can  be  used  to 
determine  the  probability  of  rejecting  H   when  the  alternative 
y.  =  6.  ;  for  i  =  1,  2,  ...,  13   is  true.   The  test  procedure  is 
known  to  have  the  desirable  property  that  the  probability  of  re- 
jecting H   when  it  is  false  is  greater  than  a;  that  is,   H   is 
more  likely  to  be  rejected  when  it  is  false  than  when  it  is  true. 

3 . 5  EXAMPLE 

For  brevity,  suppose  that  the  number  of  cogs  is  three  and 
let  the  following  table  give  the  values  determined  from  the  random 


samples  and  the  simulator, 


i 

Xi 

s2 

1 

'  ! 

n. 

l 

\ 

1 

20 

16 

48 

21 

2 

32 

4 

48 

32 

3 

24 

16 

48 

26 

TABLE 

1 

The  hypothesis  being  tested  is: 


vs 


HQ;  M1  -  Xiy   y2  =  X2,  and  y3  =  A3 


Hl:   Ul  ^  ^1'   °r  y2  ^  ^2  °r  y3  ^  X3 


Let  the  level  of  significance  be  .05.  The  number  x  r>c(3)  is  found 
from  chi-square  tables  to  be  7.815.  The  value  of  the  test  statistic 
D  calculated  from  the  given  values  is: 

.  48 (-1)2  +  A8i0)   48(-2)2  _ 

16       4       16      D 

Since  the  value  of  the  test  statistic  D  exceeds  the  value  x""ac(3) 


2 
.05 


the  null  hypothesis  is  rejected  at  a  .05  level  of  significance. 

3.6  DISADVANTAGES  OF  TEST  1 

The  example  points  out  two  disadvantages  of  this  test.   When 
the  test  statistic  rejects  the  null  hypothesis,  nothing  is  revealed 
about  the  validity  of  the  individual  simulator  values.   It  is  pro- 
bably true  that  some  of  the  simulator  values  are  sufficiently  accurate 

even  when  the  null  hypothesis  is  rejected.   In  fact,  the  data  in  the 

—  2 

above  example  shows  that  X_  »  X_   and   S„   is  small,  so  that  the 


simulator  value  for  cog  2  is  obviously  good.   It  seems  desirable 


that  the  statistical  procedure  not  only  test  the  hypothesis,  but  also 
locate  those  cogs  on  which  further  work  need  be  done. 

Another  disadvantage  revealed  by  the  example  is  that  each  of 
the  simulator  values  is  within  ten  percent  of  the  sample  means,  but 
the  null  hypothesis  is  rejected.   Since  a  ten  percent  error  is  pro- 
bably a  tolerable  amount  as  far  as  validating  the  simulator,  perhaps 
the  hypothesis  should  be  stated  as: 

Hq:   (1-p)  \±   <:  \i±  <Z    (1+p)  X±  i  =  1,  2,  ...,  13 

vs 

H':   y.  <  (1-p)  X.      or  y  .  >  (1+p)  X,  some  i 

1    i      r   i       i      r   i 

where  0  £  p  ^  1   is  a  permissible  fraction  of  error.   The  second 

statistical  test  considers  such  a  null  hypothesis. 

A.   INDIVIDUAL  STATISTICAL  TESTS  FOR  COMPOSITE  TWO-SIDED  HYPOTHESES. 

4 . 1   BACKGROUND 

Two  disadvantages  of  the  first  test  procedure  were  pointed 
out.   The  test  developed  in  this  section  eliminates  both  of  those 
disadvantages.   Consider  testing  each  cog,  individually.   A  test 
statistic  will  be  derived  for  each  cog,  and  individual  values  of 
cog  MSRT  are  either  validated  or  invalidated.   Let  k  be  any  one 
of  the  13  cogs.   The  statistical  hypothesis  tested  in  this  section 
is: 


vs 


Hr  "k  *  1    or   uk  *  %■ 
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For  S   applications   6.   and   6>  would  probably  be  taken  to  be 

(1-p)  A,   and   (1+p)  A,  ,   respectively.   This  tests  the  hypothesis 

that  the  true  mean  differs  from  the  simulated  mean  by  no  more  than 

a  fraction  p. 

Although  attention  is  restricted  to  a  single  cog  at  a  time, 

the  fact  that  the  statistical  hypotheses  are  two-sided  and  composite 

complicates  the  problem.   Nevertheless,  a  test  can  be  derived  which 

has  "optimal"  properties.  Lehmann  [2]  has  shown  that  the  test  which 

will  be  developed  is  the  uniformly  most  powerful  unbiased  test  of 

size  a.   It  is  by  this  criterion  that  the  test  is  considered  best. 

4.2  DESCRIPTION  OF  THE  TEST 

Since  the  distributions  of  the  populations  are  unknown,  it  is 
necessary  to  assume  that  the  sample  sizes  are  sufficiently  large  so 
that  the  sample  variances  can  be  substituted  for  the  unknown  popu- 
lation variances  in  the  test  procedure.   The  large  sample  sizes  are 
also  needed  to  justify  appealing  to  the  central  limit  theorem  to 
take  the  distribution  of  X,   to  be  normal  with  mean  y,   and  variance 

ak '  v 

Let  R,   be  that  subset  of  the  sample  space  containing  all  of 
those  points   (X.,  ,X~, . . .  ,X  )   such  that  their  sample  mean  X  is 
either  greater  than  a  constant  L,   or  less  than  a  constant  U,  ;  i.e., 

Rk  =  {(X1,...,Xn)  |  X  <  Lk  or  X>Uk>. 

The  uniformly  most  powerful  unbiased  test  of  size  a  has  a  particularly 
simple  form  with  the  set  R,   as  the  rejection  region.   Thus,  if 
(X,  ,X  «, . . .  ,X,   ),  the  vector  of  sample  response  times  for  cog  k, 


is  in  the  set  R^  reject  H  .   Otherwise,  accept  H  .   Hence,  all 
that  remains  to  specify  completely  the  best  test  is  to  find  the 
values  of  the  constants  L,   and  U,  .   These  are  determined  from  the 
conditions  which  state  that  the  probabilities  of  rejecting  H   when 
y,  =  (l-p)  A,   and  y,  =  (1+p)  A,   must  both  be  a  in  order  that  the 
test  be  unbiased  of  level  a.   That  is, 

P[*k  <  Lk  °r  \  >  Uk  I  \   =  (1_p)  \]    =  a 
and 
4.2.1      P[Xk  <  Lk  or  Xk  >  Uk  |  yk  =  (1+p)  Xfe]  =  a 

Since  the  distribution  of  X.   is  symmetric  and  y,   is  a  location 
parameter  for  that  distribution,  it  is  easily  shown  that 
L  =  A,  -  C   and  U,  =  A,  +  C,  .   Because  the  problem  reduces  to 
finding  the  single  constant  C,   only  one  of  the  probability  state- 
ments is  necessary.   Thus,  for  a  given  level  a  equation  4.2.1  can 
be  used  to  uniquely  determine  C,  . 

By  standardizing  the  random  variable  X,   (using  S,  A/n,   in 


place  of  a,  /-y/n  )   equation  4.2.1  can  be  replaced  by  the  equivalent 


statement 


4.2.2 


-yp  v    ,  ck-p  xk 
sk  'fa      sk  /y\ 


=  1  -  a, 


where  Z   is  standard  normal  random  variable,   N(0,1).   Let 

F(y)=P[Z£y],      ak=yn^pAk/Sk     and     \  =  J\  \   /    \- 

To  determine  the  constant  C,   or  equivalently  the  constant  bk> 

the  zero  of  the  equation: 

4.2.3  F(bk-ak)  -  F(-bk-V  -  1  -  a 
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must  be  determined.   Some  one  dimensional  search  procedure  is  needed 
to  find  that  value  since  an  explicit  solution  is  unknown.   Fortunately, 
in  many  cases  the  search  is  unnecessary  for  F(~b,  -a,  )   is  approximately 
zero.  When  this  is  the  case,  standard  normal  distribution  tables  can 
be  used  to  look  up  the  value  of  b,   such  that 

4.2.4  FO^-a^  =  1  -  a. 

4.3  OUTLINE  OF  PROCEDURE 

1.  Choose  a  level  a  and  a  fraction  p  and  calculate 

X.  ,  S.  ,  and  a.   for  each  cog  k. 
k   k         k  ° 

2.  Using  the  standard  normal  distribution  tables  find 
the  value  of  b,   such  that  F(b, -a  )  =  1  -  a. 

3.  Calculate  F(-b,-a,).   If  this  value  is  sufficiently 
small  (.0005  should  be  small  enough),  take  b,   to 
be  the  value  determined  in  step  2  and  proceed  to 
step  6. 

4.  If  F(-b,-a,  )  >  0.0005,  choose  a  new  value  b'  >  b,. 

5.  If  the  value  of  F(b'-a  )  -  F(-bk~a  )  >  1  -  a 
decrease  b'   and  try  again.   If  the  value  is  less 
than  1  -  a  increase  b'   and  repeat  this  step. 

6.  When  the  value  of  F(b'-a  )  -  F(-b'-a^)  =  1  -  a,  C 

is  then  b'  Sk  /yv  Lk  =  Xk  -  Ck  and  UR  -  Xfc  +  Ck- 

7.  If  X,  >  U,   or  X,  <  L,   reject  H  .   Otherwise 
accept  H 


11 


4.4   EXAMPLES 

A.     1.   Test  the  hypothesis  that  the  mean  supply  response 

time  for  cog  1  is  within  10%  of  the  simulator  MSRT. 


H  :    0.90  An  £  y_  £  1.10  A. 
o  11        1 


vs 


H1:    \i±   <  0.90  \       or  y   >  1.1  A 

Let  the  level  of  significance  be  a  =  .05.   The  sample 
mean  and  the  sample  standard  deviation  based  on  a 
random  sample  of  size  100  are  found  to  be  X  =  38 
and   S  =  5.   The  simulator  MSRT  is  A,  =  35  and 
a,      is  calculated  to  be  7. 

2.  From  the  standard  normal  tables  find  b,  -  7  =  1.645 
and  thus  b-  =  8.645. 

3.  Since  b  +  a  ■  15.645  no  iterative  search  need  be 
made;   (^  =  4.3225,  L±  =   30.6775  and  U±   =  39.3225. 

4.  Since  X   is  in  the  interval  [30.6775,  39.3225]  the 
null  hypothesis  is  not  rejected. 


B.     1.   Test  the  same  hypothesis  as  above,  but  suppose  the 
mean  and  standard  deviation  calculated  from  a 
random  sample  of  size  36  are  X.  ■  38  and  S..  =  35. 
Let  the  simulator  MSRT  still  be  A..  =   35  and  calculate 
a..   to  be  0.60. 
2.   From  standard  normal  tables  find   b,  -  0.60  =  1.645, 
and  thus  b,  =  2.245. 
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3.  Since  F(-b  -a  )  =  .00225,   choose  a  new  value  bj  >  b  . 
Suppose  the  new  value  is   b'  =  2.280. 

4.  The  value  of  FCb^-a^  -  FC-b'-a.)  =  F(1.680)  -  F(-2.880)  = 
0.9535  -  0.0020  =  0.9515  which  is  now  larger  than  1  -  a. 
Choose  a  new  value  b'  <  2.280,  say  b'  =  2.265. 

5.  The  value  of  F(bJ-a  )  -  F(-t>J-a  )   is  now  0.9520  -  0.0021 
=  .9499,  or  within  .0001  of  the  value  of   1  -  a,   and 

the  iterative  search  is  terminated. 

6.  Calculate  C  —  13.21,  L  =  21.79  and  IL  =  48.21. 

7.  Since  X   is  in  the  interval  [21.79,  48.21]  the  null 
hypothesis  is  not  rejected. 

Examples  A  and  B  illustrate  the  simple  procedures  which  must  be 
followed  in  order  to  test  the  hypothesis  that  the  true  population  mean  is 
within  a  given  percentage  of  the  simulator  MSRT's.   For  the  data  which 
is  to  be  tested  in  the  Ships  Supply  Support  Study  it  would  probably  rarely 
be  necessary  to  perform  an  iterative  search  for  the  constants  C,  .   In 
example  B,  the  sample  standard  deviation  was  forced  to  be  unrealistically 
large  in  order  to  present  a  case  where  a  search  was  necessary.   Even  if 
a  search  were  necessary,  a  trial  and  error  procedure  by  hand  quickly  zeros 
in  on  the  solution. 

4.5  AN  EXTENSION  OF  TEST  2 

Whenever  the  null  hypothesis  stating  that  the  population  mean  supply 
response  time  is  within  a  given  percentage  of  the  simulator  MSRT  is  rejected, 
it  seems  desirable  to  know  how  poor  is  the  simulator  value.   The  test  con- 
sidered in  this  section  can  be  used  to  provide  additional  information 
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when  the  hypothesis  is  rejected.   Instead  of  stating  only  that  the  hypo- 
thesis is  rejected  at  a  given  level  for  a  given  value  of  p ,  we  can  also 
determine  the  smallest  value  of  p   for  which  the  hypothesis  would  be 
accepted  at  the  given  level.   For  example,  it  may  be  the  case  that  the 
hypothesis  stating  that  the  population  mean  supply  response  time  is  within 
10%  of  the  simulator  MSRT  at  the  .05  level  is  rejected,  but  the  hypothesis 
would  be  accepted  at  the  same  level  if  p  were  0.15.   The  magnitude  of 
the  minimum  value  of  p  needed  to  accept  the  null  hypothesis  should  pro- 
vide some  information  as  to  the  amount  of  additional  work  needed,  if  any, 
to  be  satisfied  that  the  simulator  is  generating  reasonable  numbers. 

4.6  DETERMINING  THE  MINIMUM  VALUE  OF  p 

We  have  seen  that  the  null  hypothesis  is  accepted  at  a  given  level 

a  provided  that    A,  -  C,  £  X,  £  _  A ,+  C,,  where  C,   depends  on  the 

value  of  p.   We  use  this  to  determine  the  minimum  value  of   p  needed  in 

order  that  the  acceptance  interval  contain  the  number  X,  . 

r  k 

1.  Take  C,   to  be  the  absolute  value  of  the  difference 

k 

between   A    and  X  ;  that  is, 

C,  ■  |  _  A  -  X,  I  . 

2.  Determine  b,  =  C,  /nT/S,  . 

3.  Since  p   is  unknown,  a,      must  be  determined  iteratively. 

As  an  initial  trial  take   a,  =  b,  -  Z  . 

k    k    a 

4.  Calculate  F(-b, -a,  )  .   If  this  value  is,  say,  less  than 
0.0005,  the  minimum  value  of  p   is 

\   Sk 
Pmin  TV  Ak  ' 
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If  the  value  of   F(-b,-a,  )   Is  too  large  to  be  neglected, 
the  value  of   a,   must  be  determined  by  using  an  iterative 
search  just  as  the  value  of  b,   was  determined  previously. 
5.   If  F(-b, -a,  )  >  0.0005,  choose  a  new  value  a,  '  <  a,  . 

6.    if  FCbi,-3^')  *■  F^bk*^k'^  < 1  ~ a  decrease  ak'    and  tvy 

again.   If  the  value  is  greater  than  1  -  a   increase  a,  ' 
and  repeat  this  step. 
7.  When  a  '   is  such  that  F(b,-a  ')  -  F(-b  -a  ')  =  1  -  a, 

Vsk 

min    A.  /nT 
•  kv  k 

The  following  example  illustrates  the  above  calculations.   Let  us 
find  the  minimum  value  of   p  which  will  cause  the  null  hypothesis 

V  •  Ai~  *Ai*  vi*-  Ai+  pAi 

to  be  accepted  at  a  .05  level  of  significance  where   A  =  35.   The  sample 

mean  and  sample  standard  deviation  based  on  a  sample  of  size  100  are 

X  =  Al  and  S:  =  5.   We  find  that  C-  -  |  41  -  35  |  =?  6  and  l^  - 

Cn  v/n7  /  Sn  =  12.  For  the  initial  trial  we  take  an  =  b,  -  Z  =  12.0  - 
1  V  1    1  11a 

1.645  =  10.355.   Since  -b-j-a  =  -22.355,  F(-b1~a1)   is  negligible,  and 

we  take  a.   to  be  10.355.   The  minimum  value  of   p  for  which  the  null 

hypothesis  is  accepted  is  therefore 

m   (10.355) (5)  _ 
Pmin    (10) (35)      u.i^/y. 

As  with  the  search  for  b,   in  the  hypothesis  testing  problem 
with  the  given  value  of  p  an  iterative  search  will  rarely  be  necessary 
in  the  above  procedure.   In  almost  all  problems  of  practical  interest  a, 
will  be  approximately  b,  -  Z   since  F(-a,-b,  )  will  be  negligible. 
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4.7   THE  COMPUTER  PROGRAM 

Although  the  calculations  needed  to  carry  out  the  tests  in  this 
section  are  quite  simple  and  a  trial  and  error  search  usually  zeros  in 
very  rapidly  on  the  values  of  b,   or  a  ,  it  may  not  be  feasible  to 
perform  a  large  number  of  tests  manually.   For  that  reason  a  computer 
program  utilizing  a  first-order  Newton  search,  whenever  a  search  is  ne- 
cessary, is  included  as  Appendix  A.   The  program  only  requires  the  user 
to  input  the  number  of  tests,  a  value  for  p,  the  significance  level,  and 
a  tolerable  error  which  is  used  to  terminate  the  search.   (An  error  of 
0.0005  was  used  in  the  examples.)   In  addition,  the  user  must  input  the 
sample  size,  the  sample  mean,  the  sample  standard  deviation,  and  the 
simulator  MSRT  for  each  cog.   The  comment  cards  in  the  program  should  make 
the  program  self  explanatory. 

For  each  cog  the  program  determines  the  lower  and  upper  limits  of 
the  acceptance  region  for  the  given  value  of  p,  and  it  tells  if  the  hypo- 
thesis is  accepted  or  rejected.   If  an  hypothesis  is  rejected  the  program 
gives  the  minimum  value  of  p  necessary  to  accept  the  hypothesis.   Included 
with  the  program  in  Appendix  A  is  a  set  of  sample  input  and  sample  output. 

5.   SUMMARY  AND  CONCLUSIONS 

Two  tests  have  been  investigated  for  use  in  testing  the  validity 

4 
of  the  S   simulator.   The  single  assumption  necessary  for  the  procedures 

to  be  valid  seems  easily  justified  based  on  the  sample  data  which  was 

available.   Both  of  the  tests  presented  are  quite  simple  and  could  easily 

4 
be  performed  by  the  group  at  FMSO  or  some  other  member  of  the  S   team  in 

only  a  few  hours. 


16 


Because  of  the  disadvantages  of  Test  1  which  were  pointed  out  in 
Section  3.5,  Test  2  is  recommended.   Theoretically,  it  is  the  best  test 
possible  for  testing  the  given  hypothesis,  and  the  included  computer  pro- 
gram will  perform  a  large  number  of  tests  in  a  few  seconds.   A  caution 
should  be  pointed  out  concerning  the  choice  of  p   for  Test  2.   When 
choosing  a  value  for  p,  one  should  be  aware  that  he  is  essentially 
declaring  a  tolerable  error  for  the  simulation  values.   It  is  necessary 
that  the  effects  of  such  errors  on  any  conclusions  arrived  at  by  S  be 
examined.   If  the  effects  are  severe  p  must  be  small. 

It  is  the  author's  intent  that  this  report  be  useful  as  a  working 

4 
paper  for  performing  the  needed  tests  for  S  and  perhaps  also  for  future 

studies. 
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APPENOIX    A 


C ****************************************************** ****Q 

c  c 

C  VALIDATION    TESTS    FOR    SSSS    SIMULATOR  C 

C  FIRST-ORDER    NEWTON    SEARCH    IS    USED    WHEN    NEEDED  C 

C  C 

f ********************************************* *************£ 

c  c 

C       DEFINITION    OF    VARIABLES  C 

C  C 

c ******************************************************** **c 

c  c 

C  NTEST  NUMBER    OF    TESTS    TO    BE    MADE  C 

C  ALPHA  SIGNIFICANCE    LEVEL    OF    THE    TESTS  C 

C  P  FRACTION    OF    ERROR    FOR    THE    TESTS  C 

C  ZALPHA  VALUE    FROM    NORMAL    TABLES    SUCH    THAT  C 

C  P(     Z    >    ZALPHA    )    =    ALPHA  C 

C  ERROR  ERROR    TOLERANCE.       THE    ITERATIVE    SEARCH  C 

C  WILL    TERMINATE    ONLY    IF  C 

C  ABS(F(B-A)    -    F(-B-A)    -    (1-ALPHA))    <    ERROR         C 

C  AN( I)  SIZE    OF    SAMPLE    I  C 

C  XBAR(I)  MEAN    OF    SAMPLE     I  C 

C  SU)  STANDARD    DEVIATION    OF    SAMPLE    I  C 

C  SMEAN(I)  SIMULATOR    MEAN    FOR    COG    I  C 

C  PMIN  MINIMUM    VALUE    OF    P    FOR    WHICH    HYPOTHESIS  C 

C  IS    ACCEPTED  C 

C  L  LOWER    LIMIT   OF    THE    ACCEPTANCE    REGION  C 

C  U  UPPER    LIMIT    OF    THE    ACCEPTANCE    REGION  C 

c  c 

c** ************************************************ ****** **q 
DOUBLE    PRECISION    IDEC1,IDEC2 
DATA    IDEC1/1    ACCEPT    ,/,IDEC2/»    REJECT    •/ 
DIMENSION    AN(20),    S(20),    XBAR(20),     SMEAN(20) 

Q ****************************************************** ****£ 

c  c 

C     READ  INPUT  C 

C  READ  NO.  OF  TESTS*  ALPHA,  P,  ZALPHA,  AND  ERROR           C 

C  READ  SAMPLE  S  I  ZE, XBAR , STD.  DEV.,  AND  SIMULATOR  MEAN  FOR  C 

C  EACH  TEST.  C 

C  C 

Q ****** ****** ********************************************  **Q 

RE AD (5,1)     NTEST,ALPHA,P, ZALPHA, ERROR 
1    FORMAT(I5,F5.3,3F10.5) 

READ(5,3)     (AN( I),XBAR( I),S( I),SMEAN( I),     1=1, NTEST) 
3    FORMAT(8F10.0) 
C********* ********************************************* ****c 

C  C 

C  SET    UP    OUTPUT    TABLE  C 

C  C 

c** ******************************************* *************c 

WRITE(6,55)       ALPHA,    P 
55     FORMAT! '0«,6X, ******    VALIDATION       TESTS*1 
1,  »****»  ,///,•  HYPOTHESIS    TESTED:     (l-P)LAMBDA    <    MU  • 

2,»    <    ( 1+PJLAMBDA" ,//,5X, 'LEVEL    OF    SIGNI F I CANCE=    •  , 
3F5.3,5X, »P=» ,F6.3,///) 

WRITE(6,50) 

50  F0RMAT(3X,58(1H-)) 
WRITE(6,51) 

51  F0RMAT(3X,'I       N     I    DECISION    I  L  I  U  |», 
1»          XBAR                  PMIN     |» ,/,3X,58(lH-) ) 

C******* ************************************* **************£ 

c  c 

C  BEGIN    CALCULATIONS    FOR    TESTS    1    THROUGH    NTEST  C 

C  C 

c** ************************************************ ********c 

DO    40    K=l, NTEST 

ROOTN    =    SORT( AN(K) ) 

A    =    P    *    SMEAN(K)    *    ROOTN    /    S(K) 

B    =    ZALPHA    +    A 
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IFLAG    =    0 
2    XI    =    B    -    A 
X2    =    -B    -    A 

CALL    NDTR(X1,CDF1,PDF1  ) 
IFCX2.GE.    -4,0)    GO    TO    23 
CDF2    =    0.0 
PDF2    =    0.0 
GO    TO    33 
23    CALL    NDTR(X2,CDF2,PDF2) 

33  FX    =    CDF1    -    CDF2    -    (1.0    -    ALPHA) 
IF( IFLAG. EO.O)    GO    TO    34 

£*********  **************************************** A**** ****C 

c  c 

C     CAN  WE  TERMINATE  THE  SEARCH  ON  A?  C 

C  C 

c***** *********** *** ********************* ********** ********q 

IF(ABS( FXJ.LT. ERROR)  GO  TO  17 

C** ************************************************ ********- 

c  c 

C  CONTINUE     ITERATIVE    SEARCH  C 

C  >  C 

C ************************************************** ****** **c 

FPRIME  =  PDF2  -  PDF1 
A  =  A  -  FX/FPRIME 
GO  TO  2 
C* ******** ************ *******************  **************** **r 

C  C 

C     CAN  WE  TERMINATE  THE  SEARCH  ON  B?  C 

C  C 

c** ************************************** ******************c 

34  IF(ABS( FX).LT. ERROR)  GO  TO  7 

C***** ************************************************* ****c 

c  c 

C  CONTINUE     ITERATIVE    SEARCH  C 

C  C 

c***** *********************************************** ******c 

FPRIME    =    PDF1    +    P0F2 

B    =    B    -    FX/FPRIME 

GO    TO    2 
C ********************************* *************************£ 

C  C 

C  DETERMINE    ACCEPTANCE    INTERVAL  C 

C  C 

Q** **************************************************** ****r 

7    C    =    B    *    S(K)    /    ROOTN 

XMAX  =  SMEANiK )  +  C 

XMIN  =  SMEANiK)  -  C 
C ******************************************************* ***c 
C  C 

C     IS  XBAR  INSIDE  THE  ACCEPTANCE  INTERVAL?  C 

C  C 

IF(XBAR( K).GT. XMAX. OR. XBAR(K). LT. XMIN)     GO    TO    9 
C ******************************************************** **c 

C  C 

C  YES.    ACCEPT    HYPOTHESIS    AND    WRITE    ANSWERS  C 

C  C 

c** ********************************** **********************c 

WRITE(6,6)       K,IDEC1,    XMIN,    XMAX,    XBAR(K) 
6    F0RMAT(3X,«  I «  ,  13, •     I     f,A8,f     |«,       F9.3,1     |avF9«3fa     I', 
1F9.3,1     I  I') 

GO    TO    40 
C**** **************************************************** **c 

C  C 

C  NO,    REJECT    HYPOTHESIS    AND    DETERMINE    PMI N  C 

C  C 

c***** ******************************* **********************c 

9    C    =    ABSISMEAN(K)    -    XBAR(K)) 
B    =    C    *    ROOTN    /    S(K) 
A    =    B    -    ZALPHA 
IFLAG    =    1 
GO    TO    2 
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£***** *********************************************** *****#Q 

c  c 

C  CALCULATE    PMIN    AND    WRITE    ANSWERS  C 

c  c 

£** **************************************************** ****£ 

17  PMIN     =  A  *  S(K)  /  (ROOTN  *  SMEAN(K)) 
WRITE(6,18)    K,  IDEC2,  XMIN,  XMAX,  XBAR(K),  PMIN 

18  F0RMAT(3X,«  I  •  ,13,'  |  »,A8,»  |«,   F9.3,«  |»,F9.3,«  |«, 
1F9.3, •  I  ',F6.3, •  I •) 

40  CONTINUE 

WRITE(6,50) 

STOP 

END 

Q ******************************************************** **Q 

c  c 

C     SUBROUTINE  TO  CALCULATE  CDF  AND  PDF  FROM  NORMAL  DIST.  C 
C  C 

C** ************************************************** ******c 

SUBROUTINE  NDTR(X,P,D) 
AX=ABS(X) 

T=1.0/<1.0+.2316419*AX) 
D=0.3989423*EXP(-X*X/2.0) 

P=  1.0  -  D*T*( ( ( (1. 330274* T  -  1.821256)*T  +  1.781478) 
1  *T  -  0.3565638)*T  +  0.3193815) 
IF(X)  1,2,2 

1  P=1.0-P 

2  RETURN 
END 

£***** *************************************************** **Q 

c  c 

C     DATA  C 

C  C 

C** ************************************************ ********£ 

U    DATA 

6  0.05   0.1  1.645  0.0001 

100.0      48.55  12.32  36.00      75.0       63.21 

12.0      35.6  64.33  11.33      49.0       36.0 

147.      42.0  9.3  57.0       196.       48. 
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C   VALIDATION  TESTS  FOR  SSSS  SIMULATOR. 

C 

C   USES  FIRST  ORDER  NEWTON  SEARCH  WHEN  SEARCH  IS  NEEDED 

C 

DOUBLE    PRECISION    IDEC1,IDEC2 

DATA    IDEC1/*    ACCEPT    '/,IDEC2/'     REJECT    •/ 

DIMENSION    AN(20),     S(20),    XBAR(20),     SMEAN(20) 

C 

C   READ  NO.  OF  TESTS*  ALPHA,  P»  ZALPHA,  AND  ERROR 

C 

RE  ADC  5, 1  )  NTEST, ALPHA, P, ZALPHA, ERROR 

1  FORMATU  5-F5.3  ♦3F10.5J 
C 

C   READ  SAMPLE  S I ZE, X  BAR , STD.  DEV.,  AND  SIMULATOR  MEAN  FOR 
C   EACH  TEST. 

READ(5,3)  (AN( I ),XBAR( I ) , S ( I ) , SME AN( I) ,  I=1,NTEST) 
3  FORMAT (8F10.0) 

WRITE(6,55)       ALPHA,    P 
55    FORMAT ( '0» ,6X, ******    VALIDATION       TESTS*1 
l,t****t  ,///,•  HYPOTHESIS    TESTED:     (l-P)LAMBDA    <    MU ■ 

2,»    <    ( 1+P)LAM3DA» , //,5X, 'LEVEL    OF    SI GNI F I C ANCE =    », 
3F5.3,5X,,P=',F6.3,///) 
WRITE(6,50) 

50  FORMAT( 3X, 58(1H-)) 
WRITE(6,51 ) 

51  FCRMAT(3X,'|       N     I     DECISION     I  L  I  U  I  X 
1«       PMIN     I «,/, 3X, 58( 1H-) ) 

DO    40    K=1,NTEST 

ROOTN    =    SQRT( AN(K) ) 

A    =    P    *    SMEAN(K)    *    ROOTN    /    S(K) 

B    =    ZALPHA    +    A 

IFLAG    =    0 

2  XI    =    B    -    A 
X2    =    -B    -    A 

CALL  ndtr(xi,:dfi,pdfd 

IFCX2.GE.    -4.0 )     GO    TO    23 
CDF2    =    0.0 
PDF2    =    0.0 
GO    TO    33 
23    CALL    NDTR(X2,CDF2,PDF2 ) 

33  FX    =    CDF1    -    CDF2    -     (1.0   -    ALPHA) 
IF(IFLAG.EO.O)     GO    TO    34 
IF(ABS( FX) .LT. ERROR)    GO    TO    17 
FPRIME    =    PDF2    -    PDF1 

A    =    A    -    FX/FPRIME 
GO    TO    2 

34  IF(ABS( FX) . LT. ERROR)    GO    TO    7 
FPRIME    =    PDF1    +    PDF2 

B    =    B    -    FX/FPRIME 

GO    TO    2 
7    C    =    B    *    S(K)    t     ROOTN 

XMAX    =    SMEAN(K)     +    C 

XMIN    =    SMEANU)     -    C 

IF(XBAR(K) .GT.XMAX.OR.XBAR(K).LT.XMIN)     GO    TO    9 

WRITE(6,6)        K,IDEC1,    XMIN,    XMAX,    XBAR(K) 
6    F0RMAT(3X, ■ I • ,13,'      I     «,A8,»      I',       F9.3,'      |',F9.3,'      |', 
1F9.3,«     |  |«) 

GO    TO    40 
9    C    =    ABS(SMEAN(K)    -    XBAR(K)) 

B    =    C    *    ROOTN    /    S(K) 

A    =    B    -    ZALPHA 

IFLAG    =     1 

GO    TO    2 

17  PMIN  =    A    *     S(K)     /    (ROOTN    *    SMEAN(K)) 
WRITE(6,18)  K,     IDEC2,     XMIN,     XMAX,    XBAR(K),    PMIN 

18  F0RMAT(3X,«  I  '  ,13,  •     I     ',A8,'     [',       F9.3,'     \%^9.?>,%     |«, 
1F9.3,'     |,,F6.3,'     I') 

40    CONTINUE 

WRITE(6,50) 

STOP 

END 
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n    DATA 
6 

100.0 
12.0 
147. 


SUBR 
AX=A 
T  =  l. 
D  =  0. 
P=    1 

*T 
IF(X 
P=l. 
RETU 
ENO 


OUTINE 
BS(X) 
0/(1  .0 
398942 
.0-0 
-  0.35 
)  1.2, 
0-P 
RN 


0.05       0.1 
48.55 
35.6 
42.0 


NDTR(X,P,D) 

+.2316419*AX) 

3*EXP(-X*X/2.0) 

*T*(  (  (  U.330274*T    -    1.821256)*T 

65638)*T    +   0.3193815) 

2 


1.645 
12.32 
64.33 
9.3 


+    1.781478) 


0.0001 

36.00 

75.0 

11.33 

49.0 

57.0 

196. 

63.21 

36.0 

48. 
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*****   VALIDATION        TESTS   ***** 

HYPOTHESIS   TESTED:       (l-P)LAMBDA    <  MU    <   (1+P)LAMBDA 
LEVEL   OF   SIGNIFICANCE   =0.050  P   =    0.100 


N 

DECISION 

L 

U 

XBAR 

PMIN 

1 

REJECT 

30.373 

41.627 

48.550 

0.292 

2 

ACCEPT 

54.222 

70.438 

63.210 

3 

ACCEPT 

-25.135 

47.795 

35.600 

4 

REJECT  - 

18.390 

25.610 

36.000 

0.572 

5 

REJECT 

50.038 

63.962 

42.000 

0.241 

6 

ACCEPT 

40.648 

51.352 

48.000 

23 


BIBLIOGRAPHY 

[1]   Ferguson,  Thomas  S.,  Mathematical  Statistics  A  Decision  Theoretic 
Approach,  Academic  Press,  1967. 

[2]   Lehmann,  E.  L.  ,  Testing  Statistical  Hypotheses,  John  Wiley  &  Sons, 
Inc. ,  1959. 

[3]   Fix,  E.,  "Tables  of  Non-central  Chi-square,"  Univ.  of  Calif.  Pub, 
in  Statist. ,  Vol.  1,  1949,  pp.  15-19. 

[4]   Mood,  Alexander  M.  and  Graybill,  F.  A.,  Introduction  to  the  Theory 
of  Statistics,  McGraw-Hill,  1963. 

[5]   Ships  Supply  Support  Study  Group,  Second  Interim  Report,  August,  1972. 


24 


INITIAL  DISTRIBUTION  LIST 


Defense  Documentation  Center  (DDC) 
Cameron  Station 
Alexandria,  Virginia  22314 

Library,  Code  0212 

Naval  Postgraduate  School 

Monterey,  California  93940 

Library,  Code  55 

Naval  Postgraduate  School 

Monterey,  California  93940 

Dean  of  Research  Administration  (Code  023) 
Naval  Postgraduate  School 
Monterey,  California  93940 

Mr.  Herbert  J.  Lieberman 
NSUP  01A1B 

Naval  Supply  Systems  Command 
Washington,  D.  C.   20390 

Mr.  James  W.  Pritchard 
Code  NSUP  -  01A 
Naval  Supply  Systems  Command 
Washington,  D.  C.   20390 

LCDR  James  A.  Biggins,  SC,  USN 
Fleet  Material  Support  Office 
Mechanicsburg,  Pennsylvania   17055 

Professor  F.  Russell  Richards  (Code  55Rh) 
Department  of  Operations  Research 

and  Administrative  Sciences 
Naval  Postgraduate  School 
Monterey,  California  93940 


No.  of  Copies 
12 


10 


[prl  aasHflp.H 


Security  Classification 


DOCUMENT  CONTROL  DATA  -R&D 

■Security  classification  of  title,    body  ol  abstract  and  index, r-.t;  annotation  must  be  entered  when   tfic  overall  report  is   classified) 


ORIGINATING    activity   (Corporate  author) 

Naval  Postgraduate  School 
Monterey,  California  93940 


Za.  REPORT    SECURITY    C  I.  A  SSI  F  I  C  A  T  lOf 

Unclassified 


2b.  GROUP 


dEPORT  TITLE 


Validation  Tests  for  the  Ships  Supply  Support  Study 


DESCRIPTIVE   NOTES  (Type  of  report  and.inclusive  dates) 


Technical  Report 


AU  THOR(S)  (First  name,  middle  initial,   last  name) 


F.  Russell  Richards 


REPOR  T  DATE 


November  1972 


7a.     TOTAL    NO.    OF    PAGES 

29 


76.    NO.    OF    RE  FS 

5 


I.    CONTRACT    OR    GRANT    NO. 

NAVSUP   RDT&E 


9a.    ORIGINATOR'S    REPORT    NUMBERIS) 


6.    PROJEC  T    NO. 


NPS55RH72111A 


TF  38.531.001 


9b.    OTHER    REPORT    NOISI  (Any  other  numbers    that  may  be  assigned 
this   report) 


0.    DISTRIBUTION    STATEMENT 


Approved  for  public  release,  distribution  unlimited 


|.    SUPPLEMENTARY    NOTES 


12.    SPONSORING    MILITARY     ACTIVITY 


Research  and  Development  Division 
Naval  Supply  Systems  Command 


J.  ABSTRACT 


A  study  is  devoted  to  the  development  of  statistical  procedures 
to  be  used  to  test  the  validity  of  the  Ships  Supply  Support  Study  simulator. 
Some  theory  is  presented  for  each  test  procedure,  but  special  emphasis  is 
paid  to  describing  each  test  in  detail  so  that  the  tests  can  be  implemented 
by  the  project  study  group.   Examples  which  illustrate  the  required  numerical 
work  are  given  with  each  test  procedure,  and  the  advantages  and  disadvantages  . 
of  each  are  pointed  out.   A  computer  program  which  will  perform  the  calculations 
necessary  for  the  more  complicated  test  is  included. 

This  test  was  supported  by  the  Research  and  Development  Division, 
Naval  Supply  Systems  Command,  under  NAVSUP  RDT&E  No.  TF  38. 531'. 001. 


)D,F°oR:.,1473 

/N  0101 -807-681 1 


PAGE  1  ) 


Security  Classification 


Security  ClHssifiration 


KEY     WO  R  OS 


SHIPS  SUPPLY  SUPPORT  STUDY 

SIMULATION 

VALIDATION  TESTS 

STATISTICAL  HYPOTHESIS  TESTING 

MEAN  SUPPLY  RESPONSE  TIME 


LINK      A 


LINK      B 


DD  ,Tv"..1473  (  back  i 


S/N     01 Ot - 807-6BJ | 


Security  Classification 


A  -  3  I  4  o  » 


U159942 


5  6853  0 


III  iil'l   III  II   llllll 

057968  3 


